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This note delivers an entropy comparison result concerning weighted sums of i.i.d. (indepen- 
' dent and identically distributed) random variables. The main result, Theorem [U confirms a 

conjecture of Karlin and Rinott (1981). 
^ \ For a continuous random variable X with density /(x), x G R, the (differential) entropy is 

defined as 



o 



/log/, 



^ ■ and the more general a-entropy, a > 0, is defined as 



> , 

, where 

(N 



H^{X) = -^logGaiX), 

1 — a 



G^{x) = / r. (1) 



, It is convenient to define H{X) = Ha{X) = —00 when X is discrete, e.g., degenerate. (Our 

I notation differs from that of Karlin and Rinott 1981 here.) 

We study the entropy of a weighted sum, S = Yl7=i^i-^i^ i.i.d. random variables Xj, 
assuming that the density / of Xi is log-concave, i.e., supp{f) = {x : f{x) > 0} is an interval and 
log / is a concave function on supp{f). The main result is that H{S) (or Ha{S) with < a < 1) 
^ ■ is smaller when the weights oi , . . . , a„ are more "uniform" in the sense of majorization. A real 

^ . vector b = (61, . . . , 5„)^ is said to majorize a = (oi, . . . , o„)''', denoted a -< b, if there exists a 

doubly stochastic matrix T, i.e., an n x n matrix (tij) where tij > 0, '^iUj = 1, j = 1, . . . ,n, 
and tij = 1, i = 1, . . . , n, such that 

Th = a. 

A function (/)(a) symmetric in the coordinates of a = (oi, . . . , 0^)"'' is said to be Schur convex, if 

a ^ b ^ (p{a) < 0(b). 

Basic properties and various applications of these two notions can be found in Hardy et al. 
(1964) and Marshah and Olkin (1979). 

Theorem 1. Let Xi, . . . ,Xn be i.i.d. continuous random variables having a log-concave density 
on R. Then H{^^^^ aiXi) is a Schur convex function of (oi, . . . ,a„) G R". The same holds 
for HaiZti(^i^i) i/0<a<l. 
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As an immediate consequence of Theorem [H we have 



Corollary 1. In the setting of TheoremlJl subject to a fixed Y17=i entropy H(^^^i ctiXi) 

is minimized when all ai 's are equal. The same holds if H is replaced by with a E (0, 1). 

Note that Corollary [1] and hence Theorem [1] need not hold without the assumption that the 
density of Xi is log-concave. For example, if Xi ~ Gam(l/n, 1), i.e., a gamma distribution with 
shape parameter 1/n, then the equally weighted X^^L^ Xi, which has an exponential distribution, 
maximizes rather than minimizes the entropy H among X]r=i ^i-^i with Y17=i ~ more 
entropy comparison results where log-concavity plays a role, see Yu (2009a, 2009b). 

Karlin and Rinott (1981) conjectured Theorem [T] (their Remark 3.1, p. 110) and proved a 
special case (their Theorem 3.1) assuming that i) aj > and ii) f{x), the density of the Xj's, is 
supported on [0, oo), and admits a Laplace transform of the form 



where Oi > 1, (3i > 0, and < < Their proof of this special case, however, is 

somewhat complicated and does not extend easily when the additional assumptions are relaxed. 
A short proof of the general case is presented below. 

We shall make use of the convex order <cx between random variables. For random variables 
X and y on R with finite means, we say X is smaller than Y in the convex order, denoted 



for every convex function (j). Properties of <cx and many other stochastic orders can be found 
in Shaked and Shanthikumar (1994). 

Lemma 1 relates the convex order <cx and log-concavity to entropy comparisons. The basic 
idea is due to Karlin and Rinott (1981). See Yu (2009b) for a discrete version that is used to 
compare the entropy between compound distributions on nonnegative integers. 

Lemma 1. Let X and Y be continuous random variables on R. Assume X <cx Y and assume 
that the density ofY is log-concave. Then H{X) < H{Y) and Ha{X) < Ha(Y), < a < 1. 

Proof. Denote the density functions of X and Y hy f and g respectively. Note that 
because g is log-concave, EY'^ < oo, which implies H(Y) < oo as H{Y) is bounded from 
above by the entropy of a normal variate with the same variance as Y . Also, X <cx Y implies 
EX"^ < EY"^ < oo, which gives H{X) < oo. 

Using X <cx Y and Jensen's inequality we obtain 




X <cx Y, if 



E(I){X) < E(I){Y), 




H{X). 
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All integrals are effectively over supp{g) as X <cx Y implies that / assigns zero mass outside of 
supp[g) when supp{g) is an interval. 

To show Ha(Y) > Ha{X), we can equivalently show Ga{Y) > Ga{X), with Ga given by 
([T]). From the log-concavity of g and a < 1, it follows that (a — l)log(7 and hence g°'~^ = 
exp[(a — 1) log g] are convex. We may use this and X <cx Y and Holder's inequality to obtain 



Lemma [2] compares weighted sums of exchangeable random variables in the convex order. 

Lemma 2. Let Xi, i = 1, . . . ,n, be exchangeable random variables with a finite mean. Assume 
(oi, . . . , o„) ^ . . . , bn), ai, bi G R. Then 



Theorem [T] then follows from Lemmas [T] and [2] and the well-known fact that convolutions of 
log-concave densities are also log-concave. 

Remark Lemma [2] can be traced back to Marshall and Proschan (1965) (see also Eaton 
and Olshen 1972 and Bock et al. 1987). When Xj's are i.i.d., Lemma [2] is given by Arnold and 
Villaseiior (1986) for the case ai = . . . = a„ = 1/n, 6i = 0, b2 = ■ ■ ■ = bn = l/{n — 1), and by 
O'Cinneide (1991) for oi = . . . = a„ = 1/n. Further discussions and generalizations of Lemma 
[2] can be found in Ma (2000). Some recent applications of Lemma [2] in the context of wireless 
communications can be found in Jorswieck and Boche (2007). 
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